label improv data qualiti data mine multipl noisi label victor sheng foster provost panagioti ipeiroti paper address repeat acquisit label data item label imperfect examin improv lack thereof data qualiti repeat label focu improv train label supervis induct outsourc small task easier rent coder amazon mechan turk obtain expert label low cost low cost label prepar unlabel part data consider expens label present repeat label strategi increas complex show main result repeat label improv label qualiti model qualiti ii label noisi repeat label prefer singl label tradit set label cheap iii cost process unlabel data free simpl strategi label multipl time give consider advantag iv repeatedli label carefulli chosen set point gener prefer present robust techniqu combin notion uncertainti select data point qualiti improv bottom line result show label perfect select acquisit multipl label strategi data miner repertoir label qualiti cost regim benefit substanti 